--- title: Bert Visualize keywords: fastai sidebar: home_sidebar summary: "Visualize masked language modeling transformer model" description: "Visualize masked language modeling transformer model" ---
from transformers import AutoModelForMaskedLM,AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased",use_fast=True)
A piece of sample text
text = """I must not [MASK].
Fear is the mind-killer.
Fear is the little [MASK] that brings total obliteration.
I will face my fear.
I will permit it to pass over me and through me.
And when it has gone past I will turn the inner [MASK] to see its path.
Where the fear has gone there will be nothing.
Only I will remain."""
vis = MLMVisualizer.from_pretrained("bert-base-uncased")
%%time
result = predict_text(vis,text)
%%time
vis.visualize(text)